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DETAILED ACTION 
Information Disclosure Statement 

1 . The information disclosure statement filed February 28, 2002 fails to comply with 
37 CFR 1.98(a)(2), which requires a legible copy of each U.S. and foreign patent; each 
publication or that portion which caused it to be listed; and all other information or that 
portion which caused it to be listed. It has been placed in the application file, but the 
information referred to therein has not been considered. 

The list indicates certain references, but the references were not found. A 
request is made to submit the missing references. 

Specification 

2. The disclosure is objected to because of the following informalities: 
Page 3, last line: "produced" should be changed to "produce". 

Page 4, line 8: "are" should be deleted for a clear understanding of the respective 
sentence. 

Page 7, line 6, excluding the lines from the equations: The phrase "will cross 
approximately" should be corrected to "will cross approximately at the". 
Appropriate correction is required. 

Claim Rejections - 35 USC § 102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 
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4. Claims 1-12 and16-22 are rejected under 35 U.S.C. 102(b) as being anticipated 
by Roy (US Patent 6,046,763 A). 

Regarding claim 1 , Roy discloses a method for optimizing (Roy improves a 
resulting depth map by using multiple epipolar lines at col. 5, lines 6-9.) an estimate of a 
depth map (Roy obtains an estimate of a depth map by using a minimum cut graph 
method that uses epipolar lines at col. 6, lines 18-20.) of a reference image (fig. 1 1 , 
label "C subscript 0" is a view of a reference image that will be recreated from adjacent 
views C1, C2, and C3 or fig. 3, label: "DISPARITY SURFACE") through the blending of 
a plurality of depth maps (Using fig. 1 1 , Roy states that the reference image 10 can be 
constructed after stereo matching of images 11, 12, and 13 at col. 3, lines 10-14 or 
multiple epipolar lines from multiple images are combined to form a disparity surface as 
shown in figure 3, and note that "Stereo matching of images is a process of matching 
points in the different images of the same scene in order to determine the depth of the 
points, so-called depth map or disparity map, as the output of the stereo matching 
process (col. 1 , line 65 to col. 2, line 1)" or), taken two depth maps at a time (Fig. 3 
shows a disparity surface that includes multiple epipolar lines (col. 5, lines 6-9) that are 
derived from each camera of figure 13, numerals 1301 and 1302.), comprising: 
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a) calculating the reprojection energies (equation c(u,v) is a cost function 
where v is a reprojection function that includes reprojected pixel values at col. 4, lines 
29-32 and equation 3 of column 4) of assigning (Using the "6-connected" points of figure 
3, Roy states," Since the source [V] is connected to the closest points while the sink 
[T] is connected to the deepest points, the cut effectively separates the view volume 
into a foreground and background and yields the depth map of the scene (col. 6, lines 
12-16).") each of two adjacent pixels (Fig. 3 has a "6-connected" image point group) of a 
reference image (Fig. 3, label" DISPARITY SURFACE") to each of two separate depth 
maps (Fig. 3, labels "s" and T form a depth map from the 6-connected point group). 
Note that Roy states that a depth of points is a depth map, and a "6-connected" image 
point group can form a depth map consisting of 6 image points that are either assigned 
a close or deep point. And each "6-connected" image point group forms a disparity 
surface as shown in figure 3 that shows a six connected point group within the disparity 
surface. 

b) calculating the discontinuity energies (occ(u,v))associated with each pixel 
of the adjacent pixels("6-connected" image point group) of the reference image 
("DISPARITY SURFACE" of figure 3) and associated with the edge between the 
adjacent pixels of the reference image (Roy calculates occlusion discontinuity costs 
(col. 5, line 64,65 or "occ(u.v)" located last within a series of equations at col. 5, lines 
46-51.) using a cut graph (fig. 3 and from col. 5, line 16 to col. 6, line 42) that includes 
edges (col. 5, lines 28-32) and a global disparity surface or reference image. Note that 
the use of the cut graph with maximum flow of figure 3 "directly" provides the desired 
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global disparity surface or reference image at col. 5, lines 40-42 and col. 6, lines 41 ,42) ; 
and 

c) assigning depth map values for the two adjacent pixels based on a 
minimum graph cut ("capacities of C [cut] is minimized" at Roy, col. 6, line 8-10) 
between the two separate depth maps, given the adjacent pixels and the calculated 
reprojection and discontinuity energies (The cut graph utilizes the energies or matching 
cost (col. 5, lines 43,44) which was derived from the reprojected vector equation 3 of 
column 4). 

Regarding claim 2, Roy discloses the method according to claim 1 , wherein the 
step of assigning depth map values further includes: 

Adjusting the calculated reprojection energies (V or equation 3 of column 4) 
with the calculated discontinuity energies (occ(u,v) is preset to equal "reg(u,v)" as 
shown in col. 5, line 65-68). Note that reg (u,v) includes the reprojection energy V 
which is adjusted or divided by two as shown at col. 5, line 60); 

The remaining elements of claim 2 are similar to and addressed in claim 1. 
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Regarding claim 3 Roy discloses the method to claim 1, wherein the two 
separate depth maps consist of a first depth map (Roy states," When the occlusion cost 
is set to oo, the resulting disparity surface that is flat and features a single disparity value 
for the whole image" at col. 6, lines 2-4) and a second, hypothetical depth map (Roy 
states," Setting the occlusion cost to 0, each pixel of the image is independently given a 
disparity, therefore achieving maximal discontinuity in the disparity surface (col. 6, lines 
4-7).") and wherein the step of assigning depth map values includes replacing the 
depth map values of the first, estimated depth map to produce a third optimized depth 
map which then becomes the first, estimated depth map for subsequent optimization 
iterations (Roy states, "By... solving... for a disparity surface, it becomes possible to take 
[full] advantage of local coherence and improve the resulting depth map [by using the 
method above of setting the occlusion cost range from 0 to °°] (col. 5, lines 6-9)." Note 
that the disparity surface is a depth map, however, the six connected group within the 
disparity surface is a depth map of 6 image points that are being assigned a depth value 
of closer or deeper as discussed in claim 1 . 

Regarding claim 4, Roy discloses the method according to claim 3, wherein the 
second, hypothetical depth map is a complex, non-planar depth map (The hypothetical 
depth map can change from a flat surface to a curved surface based on the setting of 
the occlusion cost from 0 to ~ as discussed above and from col. 5, line 63 to col. 6, line 
7.). 
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Regarding claim 5, Roy discloses the method according to claim 3, wherein the 
two adjacent pixels constitute a neighboring pixel pair. Roy uses a six connected 
neighboring pixel pair as shown in figure 3. 

Regarding claim 6, Roy discloses the method according to claim 5, further 
including repeating (Roy calculates a difference between all points (x,y) at two depth 
values ("d" and "d+1") using "Property 1 (cut as a deth map)" at col.6, lines 21-24) the 
steps of calculating the reprojection energies, calculating the discontinuity energies, and 
assigning depth map values for each pixel pair of the reference image (Note that the 
depth or disparity values "d" and "d+1" were determined using the reprojection and 
discontinuity energy cost equation c(u, v) at column 5, lines 45-51 that assigns a flat or 
curved disparity surface as discussed) until the difference between the depth map 
values((x,y,d)-(x,y,d+1 )) assigned at each iteration of the reference image pixel pair set 
(Roy directly solves for all image pixel pair sets simultaneously by forming a global 
disparity surface therefore one iteration or calculation is needed (col. 5, lines 16,17).) 
reaches a predetermined minimum (Roy states that a minimum cut or cost can be 
achieved when the differences between pixels (x,y,d)-(x,y,d+1) results in the largest "d" 
or disparity). 



Application/Control Number: 09/891 ,344 Page 8 

Art Unit: 2621 

Regarding claim 7, Roy discloses the method according to claim 6, further 
including deriving a new second, hypothetical depth map ("desired global disparity 
surface" at col. 6, lines 41 ,42) for further processing (The processing of Roy finds the 
desired disparity surface directly without the need for further processing at col. 5, lines 
41-43 ) when the difference between the depth map values"(x,y,d)-(x,y,d+1)" assigned 
at each iteration of a reference image pixel pair (6 connected image point group of 
figure 3) set reaches a predetermined minimum (Roy states that a minimum cut or cost 
can be achieved when the differences between pixels (x,y,d)-(x,y,d+1) results in the 
largest "d"). 

Regarding claim 8, Roy discloses a method for estimating a depth map (Fig. 3, 
label:"6-connected" is a depth map of 6 points) of a reference image (fig. 3, label:" 
DISPARITY SURFACE") through the blending of a plurality of depth maps (fig. 3, 
label:"DISPARITY SURFACE" has multiple "6-connected" depth maps embedded on 
the surface of the "DISPARITY SURFACE", taken two depth maps at a time (All of the 
"6-connected" depth can be taken with two cameras as shown in figure 13, num. 1301 
and 1302), comprising: 

estimating a current depth map (Roy states that the minimum cut provides a 
depth estimate of each image point of the disparity surface or depth map at col. 6, lines 
18-20) of a specific view of a reference image (The specific view of the reference image 
will be created from alternate camera views); and 
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for each of a plurality of derived hypothetical depth maps (The derived 
hypothetical depth maps are created by setting the cost function to various values in the 
range from 0 to °°. As a result the disparity surface can be flat or contain a disparity 
surface (col. 5, line 63 to col. 6, line 7) of the reference image (Fig. 3, label: 
"DISPARITY SURFACE"), performing the following: 

for each pixel on the current depth map (x,y,d) that corresponds to a pixel 
on the hypothetical depth map (x,y,d+1 ), comparing the depth map value of the pixel on 
the current depth map with the depth map value of the pixel on the hypothetical depth 
map (Roy states that for every (x,y) depth value, the largest disparity d from the 
difference, (x,y,d)-(x,y,d+1), results in the desired disparity surface at col. 6, lines 39- 
42.) ; and 

replacing the depth map value of the pixel on the current depth map with 
the corresponding depth map value of the pixel on the hypothetical depth map (Roy 
does replace depth map values between image points (x,y,d) and (x,y,d+1) by 
determining the optimum depth map value which represents the desired global disparity 
surface from all points (x.y). Roy uses only the image points from all (x,y) to determine 
the pixel with the largest disparity. Once the pixel with the largest disparity is found, that 
pixel is used and all other pixels in (x,y) are not used because all the other pixels do not 
have the largest disparity.) if the compared depth map value of the pixel on the 
hypothetical depth map (x,y,d+1) has a higher probability of accurately representing the 
reference image (x,y,d) than does the compared depth map value of the pixel on the 
current depth map (Roy accurately finds the optimum pixel depth value or minimum cut 
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between (x, y, d) and (x,y,d+1) by default once the maximum flow is first determined at 
col. 6, line 8,9. 

Regarding claim 9, Roy discloses the method according to claim 8, wherein the 
view (fig. 3, label:" DISPARITY SURFACE") each of the plurality of hypothetical depth 
maps (x,y,d+1) includes at least a subregion (Fig. 3, label:"6-connected") of the view of 
the current depth map. 

Claim 10 is similar to and addressed in claim 4. 

Claim 1 1 is similar to and addressed in claim 6. 

Claim 12 is similar to and addressed in claim 7. 

Regarding claim 16, Roy discloses the system of claim 16 which is similar to 
claim 1 except for requiring: 

a) a first processor (preflow-push algorithm of figure 13 is used to solve the 
maximum-flow problem at col. 8, lines 28-29) calculating reprojection energies at 
numeral 1308:label "v" is a reprojection energy at col. 4, lines 29-32. 

b) a second processor (fig. 13, num. 1310) calculating the discontinuity 
energies at numeral 1310, label: "occ (u,v)" is a discontinuity energy or cost at col. 5, 
line 64-67. 

c) a third processor assigning depth map values (Fig. 13, num. 1350). 
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Regarding claim 17, Roy discloses the system according to claim 16 that is 
similar to and addressed in claim 2, wherein the third processor further includes: 

a) a fourth processor (fig. 13, num. 1310, label: "reg(u,v)" is an equation that 
is included in the reprojected energy V that is divided or adjusted by two at col. 5, line 
60) adjusting the calculated reprojection energies 

b) a fifth processor (Fig. 13, num. 1310, label c(u,v) is a cost function with "if 
constraints) determining the energy costs. 

c) a replacement device assigning depth map values (Fig. 13, num. 1330 
finds the optimal cut that separates or assigns pixels to either the source "s" or the sink 
T of figure 3.) 

Regarding claim 18, Roy disclosed the system of claim 18 that is similar to and 
addressed in claim 8 except: 

a) a first processor (fig. 13, num. 1330 is part of an algorithm) estimating a 
current depth map (Roy states that the minimum cut provides a depth 
estimate of each pixel within an image at col. 6, lines 18-20.) 

b) a second processor comprising the following: 

c) a third processor (fig. 13, numerals 1330 and 1340) comprising the 
following: 

d) a comparison device (fig. 13, num. 1330) comparing the 
depth map value (The step of 1330 finds the minimum cut 
which requires finding the largest disparity d among all pixels 
at col. 6, lines 38-42.) 
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e) a replacement device (fig. 13., num 1340) replacing the 

depth map value(This element of claim 18 was addressed in 
claim 8). 

The respective remaining elements of claim 18 were addressed in claim 8. 
Claim 1 9 is similar to and addressed in claim 1 1 . 
Claim 20 is similar to and addressed in claim 1 1 . 
Claim 21 is similar to and addressed in claim 13. 
Claim 22 is similar to and addressed in claim 14. 

Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 13,14,15 and 23 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Roy (US Patent 6,046,763 A) and in view of Anderson et al. (US 
Patent 5,179,441 A). 

Regarding claim 13, Roy suggests a Bayesian framework for stereo imaging at 
col. 1, lines 38-40. 

Anderson et al. uses a Bayesian framework for stereovision as described in the 
section "BAYESIAN ESTIMATION" starting at column 11. 
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It would have been obvious at the time the invention was made to one of ordinary 
skill in the art to modify Roy's estimation as taught in col. 6, line 18-20 with Anderson et 
al.'s teaching of a Bayesian framework, because "The application of statistical modeling 
and estimation methods has been growing in both feature-based and field-based 
approaches. The use of surface smoothness models, which is known to be effective in 
practice, is fitting into the statistical framework through a relationship to prior 
probabilities in Bayesian estimation (Anderson et al., col. 2, lines 6-11)." 

Regarding claim 14, Roy teaches the method according to claim 13, wherein the 
probability of accurately representing the reference image is determined according to 
energy costs and graph cuts (Roy uses the graph of figure 3 that includes a matching 
costs function (col. 5, lines 45-51 are a series of cost equations) and a minimum cut 
function "C" at col. 6, line 39.) 

Claim 15 is similar to and addressed in claim 8 except for requiring a Bayesian 
probability framework (This element of claim 15 was addressed in claim 13). 

However, regarding claim 15, Roy teaches the additional element of determining 
the optimum depth map value between the two depth maps ("desired global disparity 
surface" at Roy, col. 6, lines 41,42) wherein said determination is accomplished by 
minimizing the energy costs ("capacities of C [or cost] is minimized" at Roy, col. 6, line 
10) associated with graph cuts between neighboring pixel pairs (Fig. 3, "6-connected" is 
a group of adjacent image pixels within a graph cut or disparity surface). 
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It would have been obvious at the time the invention was made to one of ordinary 
skill in the art to modify the teachings of Roy's desired global disparity surface with 
Anderson et al.'s Bayesian estimation for the same reason as claim 13. 

Regarding claim 23, Roy and Anderson et al. teach a system and method that is 
similar to and addressed in claims 15 and 18 and a third processor applying a Bayesian 
probability framework: 

b) a second processor comprising the following: 

c) a third processor (Anderson et al, fig. 4, numerals 88,90 and 94 
comprise an algorithm with an Bayesian probability frame work.) applying a Bayesian 
probability framework. 

It would have been obvious at the time the invention was made to one of ordinary 
skill in the art to modify Roy's estimation as taught in col. 6, line 18-20 with Anderson et 
al.'s teaching of a Bayesian framework for the same reason as claim 13. 

Conclusion 

5. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Bonneville et al. (IEEE) is pertinent as teaching a method of applying graph cuts 
with Bayesian reconstruction of images. 

Thomo et al. (IEEE) is pertinent as teaching a method of using a maximum-flow 
minimum cut graph as shown in figure 3. 

Wu et al. (IEEE) is pertinent as using a minimum cut with edges as shown in 
figure 4. 
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6. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Dennis Rosario-Vasquez whose telephone number is 
703-305-5431 . The examiner can normally be reached on 9-5. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Leo Boudreau can be reached on 703-305-4706. The fax phone number for 
the organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 



Dennis Rosario-Vasquez 
Unit 2621 




^ LEO BOUDREAU 
SUPERVISORY PATENT EXAMINER 
TECHNOLOGY CENTER 2600 



